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ABSTRACT 


This article describes the use of decision analysis to facilitate a group 
decision-making problem in the selection of trajectories for the two spacecraft 
of the Mariner Jupiter /Saturn 1977 Project. This NASA project includes the 
participation of some 80 scientists divided by specialization among 11 science 
teams. A set of 32 candidate trajectory pairs was developed by the Project 
in collaboration with the science teams. Each science team then ordinally 
ranked and assigned cardinal utility function values to the trajectory pairs. 

The data and statistics derived from collective choice rules were used by the 
scientists in selecting the science-preferred trajectory pair. 


JPL Technical Memorandum 33-706 


vn 



INTRODUCTION 


This article describes the use of decision analysis to facilitate a group 
decision-making problem in the selection of trajectories for the Mariner 
Jupiter /Saturn 1977 (MJS77) Project. The objectives of the MJS77 Project are 
to conduct science investigations of the Jupiter and Saturn planetary systems and 
the interplanetary space between Earth and Saturn (Refs.. 1 and 2). This 
project is funded in excess of $300 million and is managed for NASA by the 
Jet Propulsion Laboratory (JPL) of the California Institute of Technology. 

NASA has selected some 80 scientists, divided by specialization among 11 
science teams, to participate on the MJS77 mission. The science investigations 
to be performed are shown in Table I. The scientists interface with the 
Project through a Science Steering Group (SSG) composed of the team leaders 
of the science teams. At the time this decision analysis was performed, 10 
of the 11 science teams had been selected. 

Two MJS77 spacecraft will be launched by NASA in August and September 
1977 on a pair of trajectories that will swing by Jupiter in 1979, encounter 
Saturn in late 1980 or early 1981, and then escape the solar system (Fig. 1). 

The spacecraft design is based on the Mariner experience, augmented in 
capability to meet the requirements for long-life, long-range communications, 
precision navigation, solar independent power, and support of the science 
investigations (Fig. 2). The spacecraft will be launched by a Titan III-E/ 
Centaur D-1T plus a solid-rocket kick stage to obtain the necessary launch 
energy. 

The selection of the trajectories for the mission is a major Project 
decision, since the trajectory characteristics will significantly affect the 
science investigations. At a meeting on October 22 and 23, 1973, a pair of 
trajectories was selected and recommended by the SSG for incorporation as 
the Project standard trajectory pair. While this trajectory pair may not 
actually be flown, nevertheless it represents a commitment on the part of the 
scientists, because the Project systems will be designed to this standard 
trajectory pair. Thus the selection of the standard trajectory pair was viewed 
as an important milestone, both by the Project and by the scientists. 

A decision analysis was performed prior to the October meeting of the 
SSG to facilitate the trajectory pair selection process. Because of the 
requirement that the trajectory pair recommended by the SSG should be based 
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Table I 




The MJS77 Science Investigations 


SCIENCE TEAM 

ABBREVIATION 

PRIMARY MEASUREMENTS 

RADIO SCIENCE 

RSS 

PHYSICAL PROPERTIES OF ATMOSPHERES AND IONOSPHERES. PLANET AND 
SATELLITE MASSES, DENSITIES, AND GRAVITY FIELDS. STRUCTURE OF SATURN RINGS 

INFRARED 

RADIATION 

IRIS 

ENERGY BALANCE OF PLANETS. ATMOSPHERIC COMPOSITION AND TEMPERATURE 
FIELDS. COMPOSITION AND PHYSICAL CHARACTERISTICS OF SATELLITE SURFACES 
AND SATURN RINGS 

IMAGING 

SCIENCE 

ISS 

IMAGING OF PLANETS AND SATELLITES AT RESOLUTIONS AND PHASE ANGLES 
NOT POSSIBLE FROM EARTH. ATMOSPHERIC DYNAMICS AND SURFACE STRUCTURE 

PHOTOPOLARIMETRY 

PPS 

METHANE, AMMONIA, MOLECULAR HYDROGEN, AND AEROSOLS IN ATMOSPHERES. 
COMPOSITION AND PHYSICAL CHARACTERISTICS OF SATELLITE SURFACES AND 
SATURN RINGS 

ULTRAVIOLET 

SPECTROSCOPY 

uvs 

ATMOSPHERIC COMPOSITION INCLUDING THE HYDROGEN TO HELIUM RATIO. 
THERMAL STRUCTURE OF UPPER ATMOSPHERES. HYDROGEN AND HELIUM IN 
INTERPLANETARY AND INTERSTELLAR SPACE 

COSMIC RAY 
PARTICLES 

CRS 

ENERGY SPECTRA AND ISOTOPIC COMPOSITION OF COSMIC RAY PARTICLES AND 
TRAPPED PLANETARY ENERGETIC PARTICLES 

LOW ENERGY 
CHARGED PARTICLES 

LECP 

ENERGY SPECTRA AND ISOTOPIC COMPOSITION OF LOW ENERGY CHARGED 
PARTICLES IN PLANETARY MAGNETOSPHERES AND INTERPLANETARY SPACE 

MAGNETIC FIELDS 

MAG 

PLANETARY AND INTERPLANETARY MAGNETIC FIELDS 

PLASMA PARTICLES 

PIS 

ENERGY SPECTRA OF SOLAR-WIND ELECTRONS AND IONS, LOW ENERGY 
CHARGED PARTICLES IN PLANETARY ENVIRONMENTS, AND IONIZED INTERSTELLAR 
HYDROGEN 

PLANETARY RADIO 
ASTRONOMY 

PRA 

PLANETARY RADIO EMISSIONS AND PLASMA RESONANCES IN PLANETARY 
MAGNETOSPHERES 

PLASMA 

WAVES* 

PWS 

ELECTRON DENSITIES AND LOCAL PLASMA WAVE-CHARGEO PARTICLE 
INTERACTIONS IN PLANETARY MAGNETOSPHERES 


* 


NOT SELECTED BY NASA AT THE TIME OF THIS STUDY 





Figure 1. 


Heliocentric View Showing the Selected Trajectory Pair (JSI and JSG). 
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The MJS77 Spacecraft. 
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on a consensus, this analysis was guided by the principles of certain collective 
choice rules, rather than by principles assuming a single decision-maker. 

This analysis provides the only example of the use of these formal concepts of 
collective choice in actual decision-making for a significant, real-world 
situation of which the authors are aware. 

The plan of this article is as follows. Following this introduction, 

Section II reviews the analyses performed for trajectory selection for previous 
Mariner projects. Section III describes the MJS77 trajectory characteristics. 
Section IV presents an overview of the MJS77 trajectory selection process. 

Section V describes the development of the candidate trajectory pairs by the 
Project and the science teams. Section VI describes the evaluation of the 
candidate trajectory pairs by the individual science teams. Section VII describes 
the collective choice analysis performed by the Project. Section VIII presents 
the deliberations of the scientists at the SSG meeting. Section IX presents a 
post-selection evaluation by the scientists of the trajectory pair selection 
process. The conclusions are summarized in Section X p 

II, PREVIOUS SELECTION PROCESSES FOR MARINER TRAJECTORIES 

All previous Mariner Projects have considered science requirements in 
the selection of trajectories for the missions. The earliest Mariner flights 
(Mariner 2 encountered Venus in 1962 and Mariner 4 encountered Mars in 
1965), with their relatively large targeting errors and with simple science 
sequences at the encountered planet, did not require extensive analysis of 
alternative trajectories vis-a-vis science. 

The first extensive analyses of trajectory requirements for science were 
performed on the Mariner Venus 1967 Project for Mariner 5 (Ref, 3) and on 
the Mariner Mars 1969 Project for Mariners 6 and 7 (Ref. 4). In the Mariner 
Mars 1969 analysis, science "value functions 11 for the six science investiga- 
tions were constructed over the feasible trajectory space for three science 
platform slewing strategies. These value functions operationally had an ordinal 
strength of measurement. Total value functions were then constructed by 
adding the individual value functions according to three weighting schemes. 

The results of this parametric analysis were presented to the scientists, 
and the flight trajectories were then selected in a meeting between the 
scientists and the Project Manager, 
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For the Mariner Mars 1971 Project, which placed Mariner 9 in orbit 
around Mars on November 14, 1971, no formal trajectory selection analysis 
with respect to science value was documented. The mission originally consisted 
of two Mars orbiters. One of the orbiters was to be placed in a high-inclination 
(i ~ 80 deg) orbit at Mars to obtain total surface coverage. The other was to be 
placed in a moderate -inclination (i~50 deg) orbit to obtain repeated coverage 
for identification of time -varying surface features. When the first of the two 
launches failed, the second spacecraft was then targeted for a compromise 
orbit at Mars with an inclination angle of 65 deg (Ref, 5), 

The formal trajectory analysis of the type developed on the Mariner 
Mars 1969 Project was extended for the Mariner Venus/Mercury Project, 
which launched Mariner 10 on a flight to Venus and Mercury on November 
3, 1973. Ordinal science value functions were constructed over the feasible 
trajectory space for each of the seven science investigations (Ref, 6). Two 
total science value functions were then formed, one by an additive and the 
other by a multiplicative combination of the individual science value functions. 
Unity weighting schemes were used in both cases. The multiplicative science 
value function was examined to test the sensitivity of the results to large 
losses in science value by one or two investigations. As a result of this 
analysis, the launch of Mariner 10 was delayed from October 14 to November 
3, 1973, The Mercury arrival date of March 29, 1974, was also selected 
on the basis of the analysis. 

Ill, THE MJS77 TRAJECTORY CHARACTERISTICS 

The feasible trajectories that can be considered for Earth/Jupiter/Saturn 
missions are constrained by a number of astrodynamic and programmatic 
factors. Flights direct from Earth to Saturn with flight times less than six 
years are presently not possible with existing launch vehicles within the total 
dollar constraints of the Project. Opportunities to fly missions to Saturn via 
a swingby of Jupiter occur approximately every 20 years, with a significantly 
reduced flight time. The next opportunity occurs in the 1976-1980 time period, 
with the most favorable launch dates in August and September 1977 (Ref, 7). 

The requirement to proceed on to Saturn almost totally constrains the 
arrival point at Jupiter. For a given arrival date at Saturn and a given launch 
date at Earth, the arrival time at Jupiter is uniquely determined for free-flight 
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trajectories. However, a trajectory speedup or slowdown propulsive 
maneuver in the vicinity of Jupiter can provide a restricted degree of freedom 
(plus or minus a few days) to the Jupiter arrival time for fixed Earth and Saturn 
times. Such a propulsive maneuver can be used to provide time synchronization 
of the spacecraft for a Jupiter satellite encounter. Variations in the Jupiter 
arrival point of more than a few hundred kilometers transverse to the free-flight 
trajectory are prohibited because of the large post- Jupiter propulsive maneuver 
required to proceed on to Saturn. 

The Saturn arrival times are constrained at the early end by the launch 
vehicle capability and the requirement to stay outside the most intense portions 
of the Jupiter radiation belts. (Shorter flight times correspond to close 
Jupiter flybys. ) The late Saturn arrival times are constrained by programmatic 
desires to minimize the total flight time because of reliability concerns and to 
reduce the large overhead costs of maintaining an operations staff during the 
long interplanetary cruise phase. For these reasons, the 1977 launches are 
constrained to arrive at Saturn between September 1980 and the end of 1981, 

The Saturn arrival point is constrained only by the requirement not to impact 
the body, rings, or satellites of Saturn. 

Within these mission constraints virtually an infinite number of 
trajectory pairs could be generated. It was the goal of the trajectory selection 
process to identify a trajectory pair which would be most appropriate for the 
science investigations and yet be consistent with the mission constraints. 

IV. AN OVERVIEW OF THE MJS77 TRAJECTORY SELECTION PROCESS 

The management organization for the trajectory selection process is 
shown in Fig. 3. The Project Science Office was responsible for coordinating 
the efforts of the scientists and the JPL experiment representatives. The 
latter are JPL engineers, responsible for the coordination of the science 
investigations, who report to both the Project Science Office and the science 
teams. The Mission Analysis and Engineering Manager was responsible for 
providing technical direction to the JPL trajectory analysts in the generation 
of trajectories and for the decision analysis. The Project Manager would 
approve a trajectory pair that had the endorsement of the SSG and the Project 
Science Office, given the assurance of the Mission Analysis and Engineering 
Manager that all of the required mission constraints were satisfied. 
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Figure 3. The Management Organization for the MJS77 Trajectory Selection Process. 
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Guidelines for the MJS77 trajectory selection process were informally 
established through conversations between the Project Manager, the Project 
Scientist, and the Mission Analysis and Engineering Manager. These 
guidelines were: 

(1) The process should focus on obtaining a trajectory pair compatible 
with both the science requirements and the mission constraints. 

(2) The process should be compatible with the Project resources 
allocated for mission analysis. The existing Project management 
structure and science interfaces should be used. 

(3) The process should not divert the efforts of the SSG from other 
Project activities, nor should it create dissention among the SSG 
members. 

(4) The process should be conceptually simple, and any documentation 
presented to the SSG should be essentially self-explanatory. 

Originally, some consideration was given to developing a multiattributed 
utility model for the trajectory pairs, with utility -independent attributes. The 
science teams would have then been requested to assess tradeoffs between 
the attributes and to assess lotteries over at least one of the attributes, much 
in the spirit of Keeney (Ref. 8) and others (Refs. 9-11). In principle, it might 
have been possible to specify an additive or multiplicative utility function for 
each science team and with these functions to search through the trajectory 
space and identify preferred trajectory pairs. This was not done for a number 
of reasons. In the first place, it is not an easy task to identify the appropriate 
set of utility -independent attributes of a trajectory pair. Even if such sets of 
attributes exist, they may be different for the different science teams. Further, 
the level of effort required to identify these attributes for each science team 
and to construct the utility functions seemed inconsistent with the guideline 
to develop a conceptually simple process that could be presented to the SSG 
in a self-explanatory document. Also, the possibility existed that the scientists 
would be reluctant or unable to provide such detail concerning their preferences 
at this early stage in the Project. They might feel that the disclosure of this 
information, even if correctly stated, could prove to be disadvantageous in 
future negotiations with the Project for resources and in decisions affecting the 
science investigations. 
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Several of the science investigations are concerned with information that 
can only be obtained during satellite encounters. Because the spacecraft-to- 
satellite geometry can change radically over short periods of time, it was 
not practical to construct continuous utility functions over the trajectory space 
for these science teams. Therefore, it was necessary to present the science 
teams with specifically defined trajectories for their evaluation. Even further, 
it was necessary to consider trajectory pairs, since the two flights of the 
mission would not necessarily be utility-independent. Some trajectory pairs 
might be considered by a science team to provide redundant information, while 
other trajectory pairs might be complementary in providing unique opportunities. 

The process finally endorsed required the JPL engineers to develop, in 
collaboration with the scientists, a set of candidate trajectory pairs that spanned 
the range of scientifically attractive alternatives. These candidate trajectory 
pairs would then be evaluated by each of the science teams. The JPL, engineers 
would then analyze the science team evaluations and present their analysis to 
the SSG. The final trajectory recommendation would be made by the SSG, 

Figure 4 shows a flowchart for this process, with a display of the documentation 
resulting from each stage. 

V. THE DEVELOPMENT OF THE CANDIDATE TRAJECTORY PAIRS 

The difficulty of developing a set of candidate trajectory pairs stemmed 
from two factors: (1) the problem of developing trajectories which spanned the 

range of feasible alternatives, and (2) the problem of identifying trajectory 
pairs which would meet the requirements of all the science teams. A two-way 
information exchange was necessary to overcome this difficulty. The JPL 
engineers determined the characteristics and constraints of the feasible 
trajectories, and transmitted this information to the science teams (Refs. 12 
and 13). Simultaneously, the individual science teams documented their 
science investigation objectives and the performance characteristics of their 
instruments (Refs. 14 and 15), Through this information exchange and sub- 
sequent direct interaction, the most important science criteria for the 
trajectories were developed (Ref, 16): 

At Jupiter: 

(1) Penetration of the Jupiter magnetic flux tube associated with the 
Galilean satellite Io with a range to lo of less than 40, 000 km. 
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Figure 4. Flow Chart for the MO S 7 7 Trajectory Selection Process. 
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(2) Close encounter of less than 50, 000 km range with at least one 
Galilean satellite other than Io. 

(3) Occultation of the spacecraft by Jupiter with respect to both the 
Sun and Earth. 

At Saturn: 

(1) A Titan encounter of less than 50, 000 km range with both Sun and 
Earth occultation. 

(2) Multiple satellite encounters of less than 100,000 km range. 

(3) Occultation of the spacecraft by Saturn with respect to both the 
Sun and Earth. 

(4) Occultation of the spacecraft by the rings of Saturn with respect 
to both the Sun and Earth. 

(5) Escape from the solar system in the direction of the Sun's motion 
through interstellar space. 

Needless to say, the priority ranking of these general criteria varied 
from team to team. Since some of the criteria are mutually incompatible on 
any single trajectory, the majority of them can be satisfied only by considering 
a pair of complementary trajectories. Nevertheless, a strategy of achieving 
the maximum number of criteria through the pairing of complementary 
trajectories was not endorsed by all the science teams. One science team 
expressed a preference for achieving the most important criteria in a 
redundant manner on both trajectories in order to maximize the probability of 
achieving these criteria. Thus differences in preferences for trajectory pairs 
could be expected between teams, even with compatible criteria, if their 
strategies with respect to redundancy differed. 

Using these science criteria as guidelines, JPE engineers developed 
a total of 105 single trajectories (Ref. 17). The trajectories were developed 
by considering free -flight trajectories which departed from Earth on the 
required launch day and encountered Saturn on the desired arrival day. The 
trajectories were then varied plus or minus a few days at Jupiter to obtain 
the most preferred geometries with respect to Jupiter satellite encounters. 
These trajectories corresponded to the most probable launch days (the opening 
of the launch period and approximately 11 days later) and covered every 
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feasible Titan arrival opportunity at Saturn from November 1980 to September 
1981, In addition, every feasible Iapetus (the second most interesting satellite 
at Saturn) arrival opportunity was covered, as well as several multiple 
satellite opportunities at Saturn which could be retargeted to Titan as late as 
90 days prior to Saturn arrival. 

All of these trajectories were designed to comply with the major con- 
straints of the mission: launch vehicle capability, total flight time, Jupiter 

closest approach, and navigation capability. The goal was to factor into the 
trajectories all of the mission constraints, so that the selection criteria could 
be based solely on the preferences of the science teams. While this goal was 
not completely achieved, e. g. , the Project preferred arrival dates at Saturn 
prior to June 1981 for cost considerations, it was achieved for all trajectory 
pairs ranked high in the evaluation. 

From these 105 single trajectories, candidate trajectory pairs were 
then assembled by picking one trajectory corresponding to the opening of the 
launch period and one trajectory to be launched about 11 days later, A total 
of 2624 trajectory pairs could be assembled from the 105 single trajectories. 

An additional mission constraint was introduced at this point which 
reduced the possible number of trajectory pairs by roughly a factor of 2. The 
mission constraint required the Jupiter encounter dates of the two trajectories 
to be separated by more than one month and the Saturn encounter dates to be 
separated by less than five months. It is desirable to separate the two Jupiter 
encounters by more than one month to avoid overloading the data retrieval 
capabilities of the mission operations. At the other extreme, the costs of 
maintaining the mission operations in the encounter configuration for many 
months become very large. Nevertheless, a few trajectory pairs not meeting 
this mission constraint but having unique characteristics not duplicated in 
other trajectory pairs were retained for consideration. None of these were 
ultimately ranked high in the evaluation. 

The selection of the set of candidate trajectory pairs was an iterative 
process, with an initial set being proposed by the JPL engineers, and 
successive iterations with the science teams resulting in the addition and 
deletion of candidate trajectory pairs. The number of trajectory pairs that 
were current candidates at any time varied from 12 to 40. The desire to 
include trajectory pairs spanning the widest range of alternatives was 
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necessarily tempered by the requirement to keep the total number at a 
manageable level. 

A report was distributed to the science teams which contained trajectory 
information and instructions for the trajectory pair evaluation (Ref. 18). The 
trajectory information consisted of tabulated data on the 105 single trajectories 
and a set of 24 candidate trajectory pairs which had been selected in a series 
of meetings between the JPL trajectory analysts and the JPL experiment 
representatives. 

The instructions first proposed that the science teams extend the 
candidate list of 24 trajectory pairs by adding trajectory pairs constructed 
from the list of 105 single trajectories. Additional trajectory pairs were 
submitted by the science teams and were then reviewed by the JPL experiment 
representatives. In this manner, 12 additional trajectory pairs were added to 
the candidate list, and four on the original list of 24 were dropped when no 
science team expressed an interest in them. No new single trajectories were 
added to the original list of 105, The list that was finally used in the evaluation 
by the science teams contained 32 trajectory pairs, numbered 1 through 36 
with four deletions (Ref. 19). This set of 32 candidate trajectory pairs will 
be denoted as 

T = 1 1 1 , 2 , . . . ,36} - 1 6 , 12, 14, 16}}. 

VI. THE SCIENCE TEAM EVALUATIONS OF 
THE CANDIDATE TRAJECTORY PAIRS 

The instructions stated two goals for the trajectory pair evaluation by 
the science teams: "The first is to suggest procedures for trajectory pair 

examination which will assist each science team in gaining an in-depth 
knowledge (of its preferences). The second goal is to , . , provide a language 
of preference which will facilitate communication . . . between the teams. " 

As the first step in the trajectory pair evaluation, the science teams 
were requested to ordinally rank, in order of decreasing preference, the 
set T of 32 candidate trajectory pairs. The result of this step was a set of 
rankings (tj , . . . t t^) by each science team (i = 1,. .. , 10), where t^ e T 
denotes the i th science team’s most-preferred trajectory pair and t^eT its 
least-preferred trajectory pair. 
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The next part of the instructions presented the procedure for determining 
the preferences of the science teams on a cardinal scale of measurement. Car- 
dinalization of preference was desired in order to measure the strength of 
preference between trajectory pairs and also to permit the use of collective 
choice rules requiring measurements on at least an interval scale. This 
cardinalization was attained through the use of utility function values operation- 
ally determined by von Neumann-Morgenstern lotteries (Ref. 20). Since the 
trajectory pairs were not being evaluated in a risk context, in principle other 
methods of cardinalization could have been employed (Ref. 21, pp. 92-99). 

The von Neumann-Morgenstern utility theory was used because of its 
theoretical consistency, wide acceptance, and ease of implementation. 


The utility function values were generated in a two-step process. For the 
cardinalization of preferences between trajectory pairs, each trajectory pair 
t| was compared to a lottery between the most-preferred and least-preferred 
trajectory pairs. The ith_ science team was requested to assign a probability 
number p^ such that it was indifferent as to whether it received the trajectory 
pair t* for sure, or the lottery which yielded the most-preferred trajectory pair 
with probability p^ or the least-preferred trajectory pair t^ with probability 
1 - p*. The utility formula corresponding to the cardinalization lottery is 


u X (tj) = pni^tj) + U-Pj) ul ( t 32 ^ 


( 1 ) 


where u X (t/) is the utility function value of trajectory pair t* for the ith 
J J 

science team. In this manner each of the 10 science teams generated 32 

probability numbers p!, one for each of the 32 trajectory pairs. 

It was recognized that the relative strength of preference between the 
least-preferred trajectory pair and the most-preferred trajectory pair could 
vary considerably from team to team. Some science investigations are 
relatively insensitive to the trajectory geometry, and these science teams 
could be expected to be somewhat indifferent between the trajectory pairs. 
Other science investigations are much more sensitive to the trajectory 
geometry, and these science teams could be expected to express strong 
preferences for certain trajectory pairs. Thus some means of obtaining 
interteam comparability between science teams for their least-preferred 
trajectory pairs would be required. 


For this normalization each of the science teams was requested to state 
a probability number p^ such that it was indifferent as to whether it received 
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the least-preferred trajectory pair t* ^ for sure, or the lottery which yielded the 
most-preferred trajectory pair tj with probability or a IT no-data TI trajectory 
pair t ^ with probability 1-p^, The utility formula corresponding to the normal- 
ization lottery is 

u^t^) = p^uVj) + (l-p^) U l (t^). (2) 

The lf no-data IT trajectory pair t ^ corresponded to a trajectory pair for which 
the science teams would obtain no data. In the instructions this trajectory 
pair was called the "Atlantic Ocean Special, M in remembrance of the flight of 
Mariner 8, which terminated abruptly in the Atlantic Ocean, 

The scientists were then requested to calculate utility function values 
for the trajectory pairs with an equation which can be derived from Eqs. (1) and 
(2) and the utility scaling assumptions that u 1 (t^) = 1.0 and u X (t = 0.0 (Fig. 5). 
The formula for calculating the utility function values for the trajectory pairs 
thus becomes 

ul (tj) = pj + (l-pj)p^. (3) 

The instructions also included a numerical example for the trajectory 
pair evaluation, a procedure for checking the internal consistency of the 
preference assignments, a request for the science teams to list the trajectory 
pair attributes that were most relevant to their evaluation, and a short bibliog- 
raphy on decision analysis. 

The science teams were given approximately one month to carry out this 
procedure. When the evaluation data from the 10 science teams were received 
by the JPL. engineers for analysis, it was immediately evident that the normal- 
ization lottery for interteam comparison had not achieved the desired result. 

The Project Scientist believed that some of the science teams had assigned 
utility function values to the least-preferred trajectory pairs which were much 
too high, while others were much too low. An attempt to negotiate revised 
utility function values for the least-preferred trajectory pairs was only partially 
successful. The final values assigned by the science teams to the least-preferred 
trajectory pairs ranged from 0.101 to 0.800. The ordinal rankings and the 
cardinal utility function values which resulted from this negotiation are shown 
in Table II. 

Several science teams were extremely risk-averse. The duration of 
the MJS77 Project is about 10 years and may represent the only foreseeable 
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Figure 5. The Generation of the Cardinal Utility Function Values. 
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Table II 

The Science Team Ordinal Rankings and Cardinal Utility Function Values 


TRAJECTORY 

PAIR 

RANK 

RSS 

UTILITY 

RANK 

RIS 

UTILITY 

RANK 

ISS 

UTILITY 

RANK 

PPS 

UTILITY 

uvs 

RANK UTILITY 

RANK 

CRS 

UTILITY 

LECP 

RANK UTILITY 

MAG 

RANK UTILITY 

RANK 

PLS 

UTILITY 

RANK 

PRA 

UTILITY 

1 

6.0 

0.772 

28.0 

0.871 

13.5 

0.850 

22.0 

0.550 

17.0 

0.820 

24.5 

0.850 

14,0 

0.700 

15.0 

0.500 

26.5 

0.600 

19.0 

0.546 

2 

6.0 

0.772 

24.5 

0.875 

22.0 

0.750 

25.0 

0.530 

22.0 

0.790 

5.5 

0,910 

7,0 

0.630 

19.0 

0.480 

1 1.0 

0.750 

18.0 

0.547 

3 

20.5 

0.600 

14.5 

0.925 

4.0 

0.960 

28.5 

0.520 

5.0 

0,910 

28.5 

0,840 

28.0 

0.450 

27.5 

0.350 

7.0 

0,900 

24.0 

0.109 

4 

20.5 

0.600 

6.0 

0.948 

26.0 

0.700 

20.0 

0.570 

2.0 

0.970 

28.5 

0.840 

27.0 

0.480 

26.0 

0.351 

7.0 

0.900 

25.0 

0.108 

5 

20.5 

0.600 

9.0 

0.943 

28.5 

0.670 

17.0 

0,600 

4.0 

0.920 

13.0 

0.880 

6.0 

0.900 

6.0 

0.800 

1.5 

1.000 

6,0 

0.600 

7 

20.5 

0.600 

32.0 

0.750 

32.0 

0.600 

32.0 

0,500 

18,0 

0.810 

32.0 

0.800 

30.0 

0.430 

25.0 

0.360 

26.5 

0.600 

26.0 

0,107 

8 

20.5 

0.600 

20.0 

0.886 

10.0 

0.900 

5.0 

0.790 

13.5 

0.830 

21.0 

0.860 

11.0 

0.740 

7.0 

0.700 

17.5 

0.700 

9.0 

0.556 

9 

20.5 

0.600 

12,0 

0.930 

16.0 

0.800 

13.0 

0.650 

26.0 

0.760 

9.5 

0.890 

19.0 

0.610 

16.0 

0.490 

17.5 

0.700 

13.0 

0.552 

10 

3.0 

0.887 

11.0 

0.935 

24.0 

0.720 

8.5 

0,720 

19.5 

0.800 

2.0 

0.960 

20,0 

0,600 

24.0 

0.400 

17.5 

0.700 

7.0 

0.558 

n 

20.5 

0.600 

13.0 

0.928 

7.0 

0.920 

3.5 

0,810 

28.5 

0.720 

5.5 

0.910 

26.0 

0.490 

31.0 

0.260 

32.0 

0.500 

27.0 

0.106 

13 

8.0 

0.652 

18.0 

0.915 

22.0 

0.750 

28.5 

0.520 

13.5 

0,830 

31 .0 

0.810 

31.0 

0.320 

30.0 

0.280 

26.5 

0.600 

28.0 

0.105 

15 

3.0 

0.887 

29.5 

0.869 

16.0 

0.800 

18.5 

0.580 

9.0 

0.840 

17.5 

0.870 

16.0 

0.650 

20.0 

0.430 

11.0 

0.750 

22.0 

0.543 

17 

20.5 

0.600 

4.0 

0.966 

13.5 

0.850 

6.5 

0.750 

13.5 

0.330 

24.5 

0.850 

9.0 

0.770 

14.0 

0.510 

17.5 

0.700 

8.0 

0.557 

18 

20.5 

0.600 

8.0 

0,944 

10,0 

0,900 

15.0 

0,630 

22.0 

0.790 

13.0 

0.880 

24,0 

0.560 

17.5 

0.485 

26.5 

0.600 

29.0 

0.104 

19 

20.5 

0.600 

26.5 

0.873 

10.0 

0.900 

28.5 

0.520 

26.0 

0.760 

13.0 

0.880 

25.0 

0.500 

29.0 

0.300 

26.5 

0,600 

30.0 

0.103 

20 

6.0 

0.772 

10.0 

0.937 

31.0 

0.620 

22.0 

0.550 

32.0 

0.630 

21.0 

0.860 

22.0 

0.580 

27.5 

0.350 

26.5 

0.600 

16.0 

0.549 

21 

20.5 

0.600 

14.5 

0.925 

3.0 

0.970 

31.0 

0.510 

22.0 

0.790 

28.5 

0.840 

29,0 

0.440 

11.0 

0.575 

7.0 

0.900 

31.0 

0.102 

22 

20.5 

0.600 

29.5 

0.869 

10.0 

0.900 

18,5 

0.580 

31.0 

0.640 

5.5 

0.910 

17.0 

0.630 

23.0 

0.405 

26.5 

0.600 

23.0 

0.542 

23 

20.5 

0.600 

24.5 

0.875 

16.0 

0,800 

25.0 

0.530 

30,0 

0.670 

5.5 

0.910 

8.0 

0.780 

22,0 

0.408 

26.5 

0.600 

20.0 

0.545 

24 

3.0 

0.887 

17.0 

0.919 

26,0 

0.700 

28,5 

0.520 

1,0 

1.000 

28.5 

0.840 

32.0 

0.300 

32.0 

0.250 

26.5 

0.600 

32.0 

0.101 

25 

20.5 

0.600 

7.0 

0.947 

28.5 

0.670 

25.0 

0.530 

9.0 

0.840 

1.0 

0.970 

5.0 

0.960 

5.0 

0.950 

11. 0 

0,750 

1.0 

1.000 

26 

20.5 

0.600 

2.0 

0.996 

10.0 

0.900 

11.0 

0.700 

7.0 

0.850 

17.5 

0.870 

3.0 

0.980 

1.0 

1.000 

1.5 

1.000 

2.0 

0.990 

27 

20.5 

0.600 

1.0 

1.000 

30.0 

0.650 

16.0 

0.610 

3.0 

0.930 

17.5 

0.870 

1.0 

1 .000 

4.0 

0.970 | 

4.0 

0.950 

3.0 

0.980 

28 

1.0 

1.000 

23.0 

0.876 

26.0 

0.700 

14.0 

0.640 

13.5 

0.830 

24.5 

0.850 

13.0 

0.720 

17.5 

0,485 

11.0 

0.750 1 

11.0 

0.554 

29 

20.5 

0.600 

5.0 

0.957 

19.0 

0.770 

6.5 

0.750 

9.0 

0.840 

3.0 

0.930 

2.0 

0.990 

2,0 

0.990 

4.0 

0.950 

4.0 

0.970 

30 

20.5 

0.600 

22.0 

0.878 

1.0 

1 .000 

12.0 

0.690 

26.0 

0.760 

17.5 

0.870 

10,0 

0.750 

21.0 

0.420 j 

17.5 

0.700 

12.0 

0.553 

31 

20.5 

0.600 

3.0 

0.987 

5.0 

0.940 

8.5 

0.720 

6.0 

0.860 

8.0 

0.900 

4,0 

0.975 

3.0 

0.980 

4.0 

0.950 

5.0 

0.960 

32 

20,5 

0.600 

19.0 

0.890 

19.0 

0.770 

22.0 

0.550 

28.5 

0.720 

13.0 

0.880 

18.0 

0.620 

13.0 

0.570 ; 

17.5 

0.700 

14.0 

0.551 

33 

20.5 

0.600 

26.5 

0.873 

6.0 

0.930 

10.0 

0.710 

19.5 

o.aoo 

9.5 

0.890 

21.0 

0.590 

12.0 

0.573 ! 

17.5 

0.700 

15.0 

0.550 

34 

20.5 

0.600 

16.0 

0.921 

19.0 

0.770 

3.5 

0.810 

13.5 

0.830 

21.0 

0.860 

15.0 

0,670 

8.0 

0.600 

11. 0 

0.750 

17.0 

0.548 

35 

20.5 

0.600 

21.0 

0.883 

2,0 

0.980 

2.0 

0.820 

13.5 

0,830 

13.0 

0.880 

12.0 

0.730 

9.0 

0.595 : 

17.5 

0.700 

10.0 

0.555 

36 

20.5 

0.600 

31.0 

0.813 

22.0 

0.750 

1.0 

1.000 

24.0 

0.770 

24.5 

0,850 

23.0 

0.570 

10.0 

0.594 

26.5 

0.600 

21.0 

0.544 




opportunity for some of these scientists to be involved in a. planetary mission. 
Given this situation, it was very difficult for them to consider the "no-data" 
trajectory pair with a significant, nonzero probability. One science team 
responded initially that they would only accept a probability of less than 0.001 
of obtaining the "no-data" trajectory pair, so that their utility function values 
for candidate trajectory pairs covered the . . remarkable range of 1,0 to 
0.999. . . " Voicing a similar concern, the JPL experiment representative for 
a second science team stated that ", . . the utility values indicate, in my opinion, 
that the team lacks a gambling nature, not that the pairs are of approximately 
equal value to our experiment; i. e, , the utilities serve more as a group 
Rorschach test than as a useful gauge to scientific judgments. " 

The utility function values assigned to the least-preferred trajectory 
pairs which were very low could be explained in terms of gaming. The 
science teams recognized that the maximum effect on the collective choice 
rules could be obtained by biasing these utility assignments downward. By 
spreading the utility function values for the candidate trajectory pairs over the 
entire range of [0.0, 1.0] rather than, say, over [0.8, 1.0], a science team 
would obviously have more influence on a collective choice rule which 
multiplies together the utility function values of all the science teams. In 
retrospect, it probably was not appropriate to request the science teams to 
evaluate the normalization lottery, thus in effect handicapping themselves. 

Two coalitions of science teams were formed during the evaluation 
process, one consisting of the fields and particles science investigations 
(CRS, LECP, MAG, PLS, and PRA) and the other of the platform -mounted 
science investigations (IRIS, ISS, PPS, and UVS). Table III shows the results 
of a statistical test for agreement among these coalitions. Kendall's 
coefficient of concordance W (Ref. 22) uses the ordinal rankings of the science 
teams to measure agreement on a scale from 0.0 to 1.0, where 0.0 implies 
complete disagreement and 1.0 implies complete agreement. For the total 
SSG, W = 0.30, which is significant at the 0.001 level for 10 teams. For the 
fields and particles coalition, whose science objectives are mutually related — 
investigations of phenomena interrelated by Maxwell's equations of electro- 
magnetism on a cosmic scale - W = 0.63, which is significant at the 0,001 level 
for five teams. For the platform-mounted science coalition, whose science 
objectives are not as closely related, W = 0.33, which is not significant at the 
0.05 level for four teams. 
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Table III 

A Statistical Test for Agreement Within Groups 


W = KENDALL'S COEFFICIENT OF CONCORDANCE 
DEGREES OF FREEDOM =31 


GROUP 

NUMBER 
OF TEAMS 

W 

LEVEL OF 
SIGNIFICANCE 

SSG 

10 

0.30 

<0.001 

FIELDS AND 

PARTICLES 

COALITION 

5 

0.63 

<0.001 

PLATFORM- 

MOUNTED 

SCIENCE 

COALITION 

4 

0.33 

~0J 
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The coalitions were important to the science teams, both for under- 
standing which trajectory pairs were likely to receive endorsement and for 
developing new, mutually satisfactory trajectory pairs. The fields and particles 
coalition met once at the Goddard Space Flight Center, Greenbelt, Maryland, 
on October 10, 1973. The platform-mounted science coalition and Radio 
Science met once at Stanford University on October 11, 1973. There is no 
evidence that the coalitions had an undesirable effect on the selection process. 

VII. THE COLLECTIVE CHOICE ANALYSIS 

After the trajectory pair evaluation data had been received from all 10 
of the science teams, a collective choice analysis was performed at JPL. In 
selecting the science teams to participate on the mission, NASA had made no 
priority assignments, preferring that conflicts be resolved as they arose rather 
than by a preassigned rule. Thus, since no single collective choice rule could 
be invoked, the trajectory pairs were ordered according to several rules, each 
with a different underlying rationale. The collective choice rules selected 
for consideration were the rank sum rule utilizing the ordinal rankings, and 
the additive rule, the multiplicative rule, and the maximin rule utilizing the 
cardinal utility function values. 

The rank sum rule is one of the oldest and most widely used. It requires 
the calculation of the mean ordinal rank for each trajectory pair, with the 
trajectory pair achieving the lowest mean rank being most preferred. It is a 
slight variation of the Borda method, wherein each individual assigns a "mark" 
to the n alternatives (ranked from worst to best) of 0, 1, ...» n-1, and the 
winner is the alternative receiving the largest total number of "marks" (Ref. 

23). An undesirable feature of this rule is that arithmetic operations are being 
performed on ordinal data. Nevertheless, it does have the virtue of simplicity, 
it requires only ordinal responses from the science teams, and it is easily 
understood. 

The additive collective choice rule defined on the cardinal utility function 
values can be written in the general form 

10 . . 

C(t k ) = £ ^ u X (t k ) (4) 

i= 1 
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where ktT denotes a particular trajectory pair, and X 1 is a weighting factor for 
the ith science team. 

The following sufficient conditions for an additive collective choice rule 
have been given by Harsanyi (Ref. 24): 

Condition 1 ; Collective choice satisfies the postulates of utility theory 
(specifically, postulates I, II, III, and IV of Marschak 
(Ref. 25)). 

Condition 2 ; Individual preferences satisfy these same postulates. 

Condition 3 : If two alternatives are indifferent from the standpoint of 
every individual, they are indifferent from a collective 
choice standpoint. 

The intuitively appealing "reasonableness" of these conditions argues in 
favor of the additive form, Harsanyi (Ref. 24) has discussed the ethical 
justification of this form of collective choice. For comments and caveats, 
see Sen (Ref. 21, pp. 89-104, 131-151). Nevertheless, there still remains 
the difficult problem of making interteam utility comparisons. This problem 
requires consideration of the related issues of the interteam normalization 
through the scaling of each science team s utility function and the choice of 
the \ lf s, the weighting factors. 

The multiplicative collective choice rule is based on the Nash Bargaining 
Model with a restricted bargaining set of pure strategies. The Nash 
Bargaining Model is one specific interpretation of a H fair 7r solution to a 
bargaining problem (Refs. 26 and 27). The axioms of fairness postulated 
by Nash are the following: 

Axiom 1: Invariance with respect to utility transformations. 

Axiom 2 ; Pareto optimality. 

Axiom 3; Independence of irrelevant alternatives. 

Axiom 4; Symmetry. 

The Nash solution maximizes the product of the increase in the utility 
function values which the participants gain with respect to a IT status quo. 11 
The status quo is the alternative which the participants receive if they cannot 
achieve a mutually acceptable bargain. The use of the Nash Bargaining Model 
with a restricted bargaining set of pure strategies has been explored by 
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Bonnardeaux, Dolait, and Dyer (Ref. 28). The conclusion of their efforts was 
that this model could provide useful information, particularly if the status quo 
alternative is chosen appropriately. Loosely speaking, an individual given 
a comparative advantage in the status quo will maintain an advantage in the 
Nash solution to the bargaining problem. In this spirit the "no-data" 
trajectory pair was selected as the status quo for all science teams. Since for 
the "no-data" trajectory pair u 1 (t = 0.0 for all science teams, the general 
form of the multiplicative collective choice rule is 


c (t k ) = 



1 

10 


(5) 


where ke T. 


The maximin collective choice rule, discussed by Rawls (Ref. 29), 
maximizes the minimum utility function value received by any science team. 

The maximin rule can be interpreted as another definition of "fairness. " 
Although the maximin rule is simple to apply, the results are extremely 
sensitive to the interteam normalization assumptions. 

The interteam normalization procedure was one of the principal issues 
of concern in the evaluation of the trajectory pairs. The introduction of the 
"no-data" trajectory pair t ^ with u 1 (t = 0.0 for i = 1,. . . , 10 was an attempt 
to reconcile the interteam normalization problem by identifying a worst 
alternative with a common outcome for each science team. In addition, since 
the science teams were involved in the determination of the candidate set of 
trajectory pairs, it was assured that at least one trajectory pair would be 
"very good" for each science team. To this extent, the normalization 
procedure could be said to be "fair. 11 

However, because of the problem of rationalizing the results of the 
normalization lottery, two other normalization procedures were also used to 
test the sensitivity of the collective choice rules to the utility function values 
assigned to the least-preferred trajectory pairs. The second normalization 
procedure linearly transformed the utility function values of each science 
team as shown in Table II into the range [0.0, l.oj, where the value 0.0 was 
assigned to the least-preferred trajectory pair. The third normalization 
procedure linearly transformed the utility function values of each science 
team into a range assigned by the Project Scientist, based on his assessment 
of the appropriateness of the least-preferred trajectory pair for each science 
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investigation. The Project Scientist assigned the range of [o.6 f l.o] to the 
encounter -oriented science teams {RSS, IRIS, ISS, PPS, UVS, and PRA), 
and the range [o.8, l.o] to the science teams with both cruise and encounter 
objectives (CRS, LECP, MAG, and PLS). 

Another issue was the choice of the weighting factor X 1 for each science 
team in the additive collective choice rules. Two sets of weighting factors 
were used: (1) equal weights of X 1 = 1.0 for all science teams and (2) X 1 = 2,0 

for the encounter -oriented science teams (RSS, IRIS, ISS, PPS, UVS, and PRA) 
and X* = 1.0 for the other science teams (CRS, LECP, MAG, and PLS). Both 
sets of weighting factors were readily accepted by the SSG as representative 
weighting factors for a sensitivity analysis. The first set of weighting factors 
implies that all science investigations would be of equal importance to the 
mission, if they could be flown on the trajectory pair most preferred by that 
science team. No particular justification was made for the allocation of the 
second set of weighting factors, but plausible arguments would be that either 
the encounter -oriented science investigations were more important or that 
these investigations, being more sensitive to the trajectory geometry, should 
have a greater influence on the trajectory pair selection, although in principle 
the normalization lottery should have compensated for the sensitivity to tra- 
jectory geometry. 

The results of the analysis with the various collective choice rules are 
presented in Table IV. The collective choice rules were scaled to yield values 
in the range [0.0, l.o] for ease of comparison. All of the collective choice 
rules would assign a value of 1.0 to a trajectory pair which was evaluated as 
the most-preferred trajectory pair by every science team. 

The trajectory pair rankings by the rank sum rule are shown in the first 
data column of Table IV, with the values in the second column being the mean 
ranks of the science teams linearly transformed into the range (l / 3 Z , 1.0], 
with 1.0 most preferred. The next six collective choice rules are based on the 
additive form, with the two weighting factor sets times the three normalization 
procedures accounting for the six middle data columns of Table IV. Finally, 
the last two collective choice rules are based on the multiplicative rule, with 
the M no-data n trajectory pair taken as the status quo alternative. The two 
multiplicative rules differ in that the utility function values for one rule are 
scaled upward from u^t^) - p j as assigned by the science teams, and for the 
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Table IV 


The Collective Choice Rankings and Values 


LU 

V 

o 

X 

u 

LU " 

>3 

U 

LU 

—1 


RANK SUM 

ADDITIVE 

NASH 



4 

0.0 

0.6 OR 0.8 

p i 

0.0 

0.6 OR 0.8 


0.6 OR 0,8 

WEIGHTING 


1.0 

1.0 

1 

.0 

1 .0 OR 2.0 

B 





6 

u 


RANK 

VALUE 

RANK VALUE 

RANK VALUE 

RANK VALUE 

RANK VALUE 

RANK VALUE 

RANK VALUE 

RANK 

VALUE 

RANK VALUE 


31 

1 

0.822 

2 

0.887 

1 

0.724 

1 

0.901 

1 0.871 

1 

0.691 

1 

0.884 

1.5 

0.877 

1 

0.892 


29 

2 

0.797 

3 

0.875 

3 

0.692 

3 

0.884 

3 0.852 

3 

0.638 

3 

0.860 

3 

0.865 

3 

0.874 

£5 

26 

3 

0.795 

1 

0.889 

2 

0.710 

2 

0.896 

2 0.870 

2 

0.676 

2 

0.878 

1.5 

0.877 

2 

0.886 

< 

0u 

27 

4 

0.719 

4 

0.856 

4 

0.641 

4 

0.871 

4 0.833 

4 

0.596 

4 

0.848 

4 

0.839 

4 

0.856 

> 

Qd 

o 

5 

5 

0.683 

6 

0.791 

6 

0.555 

8 

0.841 

6.5 0.765 

8 

0.502 

12 

0.813 

6 

0.776 

11 

0.829 

(— 

u 

LU 

25 

6 

0.678 

5 

0.822 

5 

0.597 

5 

0.851 

5 0,800 

5 

0*535 

9 

0.822 

5 

0.804 

7 

0.836 


35 

7 

0.655 

7 

0.757 

8 

0,51! 

6,5 

0.846 

6.5 0.765 

7 

0.517 

5.5 

0.833 

8 

0.745 

6 

0.839 


17 

8 

0.622 j 

10 

0.738 

1 i 

0.475 

10.5 

0.836 

10 0.746 

10 

0.487 

8 

0.823 

10.5 

0.725 

9.5 

0.830 


6 

9 

0.611 

8 

0.755 

9 

0.488 

10.5 

0.836 

8 0.756 

11 

0.486 

10 

0.820 

7 

0.746 

9.5 

0.830 


10 

10 

0.605 

12 

0.728 

7 

0.514 

6.5 

0.846 

11 0.744 

6 

0.519 

5.5 

0.633 

14 

0.706 

5 

0.843 


CM 

LTl 












other rule from u^t^) = 0.6 or 0.8 as assigned by the Project Scientist. 

The results for the two multiplicative rules are shown in the right-hand data 
columns of Table IV. 

Since the Radio Science Team (RSS) evaluated 24 of the 32 trajectory 
pairs as "least-preferred, " the use of the interteam normalization procedure 
with u^t* ? ) = 0.0 would have given little useful information in the multiplicative 
model. Only six trajectory pairs would have received non-zero values from 
Equation 5. For a similar reason, the maximin rule of Rawls proved not to 
be useful. With the p^ normalization five trajectory pairs were tied for first 
ranking (5 , 26 , 27, 29, and 31); with the 0.0 normalization only six trajectory pairs 
were ranked (28M 0M 5>T>-2>13^all others); and with the 0.6/0.8 normalization 
only six trajectory pairs were ranked ( 10>28>1 5>1 >£ >1 3>all others). 

One well-known collective choice rule not included in the JPL analysis 
was the majority decision rule (Refs. 21 and 30), which can lead to intransitivity 
and even to violation of a weaker condition, acyclicity. A preference ordering^ 
over a set of alternatives t^e T is acyclical if and only if the following holds: 

V - t k eT: Jtj>t 2 & t 2 >t 3 & . . . & t k l >t k j->. tj>t k 

wherei- implies preference or indifference. The application of the majority 
decision rule to the 10 trajectory pairs ranked highest by the rank sum rule 
preserved acyclicity, but did result in two ties, thus forming the following 
ordering: (26>|29, 27, 3l}>25>5>8>35>17>10) with 29-27, 27-31, and 29>31. 

The trajectory pairs ranked in the top four by the majority decision rule were 
ranked in the top four by all the collective choice rules of Table IV. 


The data in Table IV indicate a substantial agreement among the nine 
collective choice rules presented there. This is partially fortuitous, partially 
a result of the statistical properties of these rules, and also a result of the 
generally compatible requirements of the science teams. All of the collective 
choice rules could be expected to be highly correlated on a statistical basis. 

Sums and products of random variables are highly correlated, even if the random 
variables are independent and uniformly distributed over their domain. For 
example, for independent random variables uniformly distributed over 
[0.0, 1.0], 1(J 

IVX; 

10 ^ 1 

i= 1 
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are correlated with a Pearson correlation coefficient of 0.89. Any positive 
correlation between the X^'s, i. e. , agreement between the science teams, 
will increase this number. For the 32 candidate trajectory pairs, the additive 
rule and the Nash rule {with normalization and X 1 = 1.0 weighting) are 
correlated with a Pearson correlation coefficient of 0.98. Agreement between 
the nine collective choice rules, as measured by Kendall’s coefficient of 
concordance (Ref. 22), is W - 0.96 for the 32 candidate trajectory pairs. 

The generally compatible trajectory requirements of the science teams 
were to a large degree assured by the detailed project planning activities of 
JFL and by the coordinated science investigation selection process of NASA. 

The MJS77 mission design has been derived from earlier work done on the 
Outer Planets Grand Tour Project (Refs. 31 and 32), and the MJS77 mission 
definition phase which followed when the Grand Tour Project was reconstituted 
as the MJS77 Project. A pre-project Science Steering Group participated in 
both of these activities (Refs, 33 and 34). At the conclusion of these activities 
NASA formally requested proposals from the science community for science 
investigations to be performed on the MJS77 mission, with the proviso that, 
if at all possible, the proposals should be compatible with the spacecraft 
design and the trajectory characteristics developed during the mission definition 
phase (Refs. 35 and 36). The degree of compatibility formed part of the 
science selection criteria. Thus it was not anticipated that the trajectory pair 
selection process would uncover major unresolvable conflicts between the 
trajectory requirements of the science teams. 

To complete the analysis of Table IV, define the relation such that 

foranym,ncT, t t if and only if t t for all r = 1, . . . , 9 where /* 

corresponds to the ordering determined by the rth collective choice rule of 

Table IV. The relation determines three equivalence classes among the 

R 

10 trajectory pairs of Table IV, such that 

{31, 29, 26} > R {27} > R {5, 25, 35, 17, 8, 10}. 

Thus, if the science teams were able to accurately express their preferences 
over the candidate trajectory pairs, and if the collective choice rules were 
appropriate, then the selected trajectory pair should be a member of the set 
{31, 29, 26}. 
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VIII. THE SCIENCE STEERING GROUP MEETING 

Following the science team evaluation and the JPL analysis of the 
candidate trajectory pairs, the SSG selected the science -preferred trajectory 
pair at the October 22 and 23, 1973 SSG meeting (Refs, 37 and 38), The afternoon 
of the first day began with a discussion of the general trajectory requirements 
of the science investigations. The JPL experiment representatives summarized 
the trajectory characteristics required by each science team, and a JPL 
trajectory analyst summarized the general compatibilities and incompatibilities 
of these trajectory requirements. After an hour it became evident that the 
trajectory pair selection could not be made on the basis of this discussion. 

While the discussion did clarify the trajectory requirements of the individual 
science teams, no rationale emerged for trading off incompatible requirements, 
and no means was found for moving from the general trajectory requirements 
to the selection of a specific trajectory pair. At this point the SSG requested 
that JPL present the collective choice analysis. 

The Mission Analysis and Engineering Manager first reviewed the 
requirements and constraints which influenced the selection of the candidate 
trajectory pairs. The JPL trajectory analyst described the trajectory 
characteristics of the trajectory pairs ranked highest by the collective choice 
rules. The Mission Analysis and Engineering Manager then described the 
rationale of each of the collective choice rules, and presented a summary 
of the collective choice analysis as shown in Tables IV and V. Table V 
shows the science team ordinal rankings for the 10 trajectory pairs ranked 
highest by the rank sum collective choice rule. 

Two observations which can be made from Tables IV and V are of primary 
importance. First, Trajectory Pairs 31, 29, and 26 are ranked in the top 
three by all the collective choice rules. In the ensuing discussion, the majority 
of the SSG expressed a preference for one of these three trajectory pairs. 

Second, the Radio Science Team (RSS) considered any trajectory pair ranked 
high by the collective choice rules to be undesirable. The Radio Science Team 
strongly preferred a trajectory pair with one trajectory which would be occulted 
at Saturn along the major axis of Saturn's rings, in this manner yielding a 
complete radio attenuation profile of the ring structure. The JPL trajectory 
analyst stated that it would be possible to improve somewhat the ring occulta - 
tion geometry without degrading the Saturn secondary satellite encounters, 
but that an optimum ring occultation and a good Titan encounter were mutually 
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TRAJECTORY PAIRS 


Table V 


The Science Team Ordinal Rankings for Preferred Trajectory Pairs 




SCIENCE TEAM ORDINAL RANKINGS 




RS5 

IRIS 

ISS 

PP5 

UVS 

CRS 

LECP 

MAG 

PLS 

PRA 

31 




3.0 

5.0 

8.5 

6.0 

8.0 

4.0 

3.0 

4.0 

5.0 

29 

2 

3 


5.0 

19.0 

6.5 

9.0 

3.0 

2.0 

2.0 

4.0 

4.0 

26 

3 

1 


2.0 

10.0 

11.0 

7.0 

17.5 

3.0 

1.0 

1.5 

2.0 

27 

4 

4 

20.5 

1 .0 

30,0 

16.0 

3.0 

17.5 

1 ,0 

4.0 

4.0 

3.0 

5 

5 

6 

20.5 

9,0 

28.5 

17.0 

4.0 

13.0 

6.0 

6.0 

1 .5 

6.0 

25 

6 

5 

20.5 

7.0 

28.5 

25.0 

9.0 

1.0 

5.0 | 

5.0 

11.0 

1.0 

35 

7 

7 

20.5 

21.0 

2.0 

2.0 

13.5 

13.0 

12.0 

9.0 

17.5 

10.0 

17 

8 

10 

20.5 

4.0 

13.5 

6.5 

13.5 

24.5 

9.0 

14.0 

17.5 

8.0 

8 

9 

8 

20.5 

20.0 

10.0 

5.0 

13.5 

21.0 

11.0 

7.0 

17,5 

9.0 

10 

10 

12 

3.0 

11.0 

24.0 

8.5 

19.5 

2.0 

20.0 

24.0 

17.5 

7.0 
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exclusive. A general concern was expressed by the SSG that any improvement 
to the ring occultation geometry should not significantly degrade the secondary 
satellite encounters. 

The team leader of the Imaging Science Investigation (ISS) expressed a 
dislike for Trajectory Pair 29, to which they had assigned an ordinal ranking 
of 19 and a utility function value of 0.770, but indicated that either Trajectory 
Pair 31 or 26 would be acceptable (utility function values of 0.940 and 0.900). 

The team leader of the Cosmic Ray Investigation (CRS) stated that they could 
accept Trajectory Pair 26 which, even though ordinally ranked at 17.5, was 
given a utility function value of 0,870 (compared to 0.900 for Trajectory Pair 
31). The team leader of the Radio Science Investigation (RSS) expressed a 
preference for Trajectory Pair 26 over Trajectory Pair 31, since Trajectory 
Pair 26 did give at least a partial ring occultation. This prompted some 
discussion on the part of the SSG to the effect that if the Radio Science Team 
really did have a preference for Trajectory Pair 26 over Trajectory Pair 31, 
then why was this not reflected in their utility function values? See Table VI 
for the RSS, ISS, and CRS ordinal rankings and utility function values supporting 
this discussion. The other team leaders expressed their satisfaction with 
either Trajectory Pair 31 or 26. 

On the basis of this discussion. Trajectory Pair 26 was tentatively 
selected as the science -prefer red trajectory pair. The JPL trajectory analysts 
worked that night to improve the ring occultation geometry of Trajectory Pair 
26 without degrading the secondary satellite encounters. This analysis was 
presented to the SSG the following morning and met with approval. Following 
the SSG meeting, one further change was made to the selected trajectory pair. 
This change improved the satellite encounters while retaining essentially the 
same ring occultation geometry. The possibility remains that had Trajectory 
Pair 31 been improved in the same manner as Trajectory Pair 26, it might 
have emerged as the science-preferred trajectory pair. 

The modified version of Trajectory Pair 26 was approved by the Project 
Manager, and was documented as the MJS77 "Standard Trajectories" (Ref. 39). 
The two individual trajectories were labeled "JSI" and "JSG, " where JS stands 
for Jupiter/Saturn, and I and G stand for the two Jupiter satellites Io and 
Ganymede, which are encountered on the corresponding trajectories. 
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Table VI 

RSS, ISS, and CRS Evaluation of Three Trajectory Pairs 


TRAJECTORY 

PAIRS 

RSS 

RANK value 

ISS 

RANK VALUE 

CRS 

RANK VALUE 

31 

20.5 

0.600 

5.0 

0.940 

8.0 

0.900 

29 

20.5 

0.600 

19.0 

0 . 770 

3.0 

0.930 

26 

20.5 

0.600 

10.0 

0.900 

17.5 

0.870 
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IX, THE SCIENCE TEAM QUESTIONNAIRE 


After the selection of JSI and JSG as the Project standard trajectory pair, 
a questionnaire on the trajectory selection process was mailed to the members 
of the SSG. Nine of the 10 science teams responded to the questionnaire. In 
some cases the questionnaire was filled out by the science team leader, in other 
cases it was discussed by the entire science team, and in still others it was 
filled out by the JPL experiment representative after consultation with the 
science team leader. 

The questionnaire contained 18 questions, each requiring a response on 
a scale from -5 to +5. Depending on the specific question, a response of -5 
corresponded to "no, ,r M not viseful, M ,! very bad, n or 'Very unfair, 11 while a 
+5 corresponded to "yes, " "very useful, " "very good, " or "very fair, " 

The responses to the questionnaire are given in Table VII, where the 
left-hand column contains an abbreviated version of the questions actually posed. 
The next column gives the median response of the science teams. The science 
teams and their responses are given on the right and identified by number 
rather than by name to preserve confidentiality. 

The responses of Science Team 3 warrants an initial comment. Science 
Team 3 strongly felt that the concept of achieving complementary objectives 
on the two trajectories was incorrect. This science team preferred two 
redundant trajectories to maximize the probability of achieving the most 
important objectives. Thus their principal objection to the trajectory selection 
process was that the wrong alternatives were being evaluated. In a letter to 
the Project Science Office they stated: 

"The . , , team feels that the current concept of two independent 
trajectories is not basic to the mission. The second spacecraft 
should be considered principally as a backup to the first until 
the success of the first is assured. The science return from 
the first spacecraft should be maximized. The utility analysis 
should be applied (only) to this first mission. " 

For most questions the inclusion of the responses from Science Team 3 makes 
no significant difference in the median response. Only in Question 9 does it 
change the median response by as much as two units. 
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Table VII 


Responses from the Science Team Questionnaire 



MEDIAN 

SCIENCE TEAMS 

QUESTIONS 

RESPONSE 

#1 - 

#2 

*3 

#4 

#5 

#6 

*7 

#8 


1. DID THE PROCESS OF ORDINAUY RANKING THE 

TRAJECTORY PAIRS AID YOUR UNDERST ANDING OF THEM? 

+5 

+5 

44 

-5 

45 

+5 

43 

+4 

+5 

+5 

2. WERE THE ORDINAL RANKINGS A USEFUL WAY TO 
COMMUNICATE YOUR PREFERENCES? 

+3 

0 

43 

-5 

+5 

*4 

-2 

44 

0 

45 

3. DID THE ASSIGNMENT OF CARDINAL UTILITY VALUES 
INCREASE YOUR UNDERSTANDING OF THE PAIRS BEYOND 
WHAT RESULTED FROM THE ORDINAL RANKINGS? 

0 

-2 

0 

-5 

-5 

+5 

44 

43 

0 

+3 

4. DID THE CARDINAL UTILITY VALUES COMMUNICATE 
USEFUL INFORMATION REGARDING YOUR PREFERENCES 
BEYOND WHAT WAS CONTAINED IN THE ORDINAL 

rankings? 

+2 

-4 

+2 

-5 

-5 

4-4 

4-3 

44 

0 

43 

5. WAS THE ASSIGNMENT OF p* USING THE "NO-DATA 11 
TRAJECTORY PAIR A USEFUL EXERCISE? 

-4 

0 

-4 

-5 

-5 

+5 

-4 

-5 

-2 

-5 

6. WERE YOUR CARDINAL UTILITY VALUES AN ACCURATE 
MEASURE OF THE 5CIENCE VALUE OF YOUR INVESTIGA- 
TION AS FLOWN ON EACH TRAJECTORY PAIR? 

+2 


4-2 

-5 

-5 

+5 

-1 

+2 

-3 

43 

7. WAS THE SELECTED TRAJECTORY PAIR GOOD OR BAD FOR 
YOUR TEAM? 

+•3 

+3 

+3 

-2 

4-3 

44 

4-3 

+5 

+2 

45 

8. WERE THE COLLECTIVE CHOICE RULES A USEFUL WAY TO 
EXPRESS GROUP PREFERENCES? 

-1 

0 

+3 

-5 

-3 

+4 

-1 

44 

-2 

-4 

9. WERE THESE COLLECTIVE CHOICE RULES AN ACCURATE 
MEASURE OF THE SCIENCE VALUE OF THE MISSION AS 
FLOWN ON EACH TRAJECTORY PAIR 7 

-2 

-5 

4-2 

-5 

-5 

4-4 

-2 

44 

-2 

43 

10. WA5 THE SELECTED TRAJECTORY PAIR A GOOD OR SAD 
DECISION IN TERM5 OF THE SCIENCE VALUE OF THE 
MISSION? 

+3 

+4 

4-3 

-2 

4-3 

44 

+3 

+4 

0 

+5 

11. WA5 "GAMING" ATTEMPTED BY MEMBERS OF THE SSG? 

+2 

- 

0 

-5 

+3 

+2 

+2 

' +3 ; 

+5 

0 

12. DID "GAMING" AFFECT THE SELECTION OF THE 
TRAJECTORY PAIR? 

0 

0 

-1 

-5 


0 

D 

D 

D 


13. DID THE COALITIONS HAVE A BENEFICIAL OR UNDESIR- 
ABLE EFFECT ON THE TRAJECTORY PAIR SELECTION? 

*2 

+3 

-2 

0 

ID 

4-2 

D 


D 

D 

1-4. WAS THE TRAJECTORY PAIR SELECTION PROCESS FAIR? 

f 4 

+4 

+2 

0 

■ 

t4 

f3 

44 

D 

4-5 

15 WOULD THE SAME TRAJECTORY PAIR HA\^ BEEN 
SELECTED WITHOUT THE DEVELOPMENT OF THE 
ORDINAL RANKINGS AND THE CARDINAL UTILITY 
VALUES? 

f2 

4-2 

1 5 

+5 

D 

D 


D 

D 

+5 

16. DID THE USEFULNESS OF THE ORDINAL RANKINGS 
AND THE CARDINAL UTILITY VALUES JUSTIFY THE 
EFFORT REQUIRED TO GENERATE THEM? 

<N O 

0 

+2 

-5 


D 


+3 


D 

17. WOULD YOU LIKE A SIMILAR ANALYSIS TO BE 
PERFORMED FOR CRITICAL MISSION EVENTS SUCH 
AS TITAN ENCOUNTERS? 

-2 

0 

-2 

-5 

-5 

4-5 

+1 

4-3 

-2 

-3 

18 WOULD YOU LIKE TO REPEAT THE ANALYSIS IN 
1977 TO SELECT THE TRAJECTORY PAIR TO BE 

launched? 

0 

0 

+1 

-5 

-5 

♦ 5 

0 

+3 

-5 

-3 
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The first two questions concerned the process of the ordinal ranking of 
the trajectory pairs by the science teams. There was almost unanimous 
agreement that the ordinal ranking process had increased the science teams’ 
understanding of the relationship between their science objectives and the 
characteristics of the trajectory pairs. In addition, they tended to agree that 
the ordinal rankings were a useful way to communicate their preferences to 
the other science teams. Although there was not much agreement, the responses 
to the two questions regarding the assignment of cardinal utility function values 
to the trajectory pairs indicated that for some science teams additional under- 
standing of the trajectory pairs resulted, and additional information was 
communicated. 

The assignment of a utility function value to the least-preferred trajectory 
pair based on a lottery between the most-preferred trajectory pair and the 
"no-data" trajectory pair was not considered to be a useful exercise. Never- 
theless, five of the science teams indicated that the utility function values 
did provide an accurate measure of the "science value" of their investigation. 

All of the science teams except Science Team 3 believed that the selected 
trajectory pair was a good one for their own team. 

The next two questions explored the usefulness of the collective choice 
rules. The science teams were not in agreement about whether these rules 
appropriately expressed group preferences, or about whether these rules 
provided an accurate measure of the science value of the mission as flown 
on each trajectory pair. They did generally agree that Trajectory Pair 26 was 
a good decision by the SSG in terms of the science value of the mission. 

Although over half of the science teams believed that "gaming" had 
occurred, in the sense of biasing stated preferences in order to influence the 
trajectory pair selection, there was no agreement as to its effects. The 
science teams generally believed that the coalitions which were formed actually 
had a beneficial effect on the trajectory pair selection. None of the science 
teams believed that the trajectory pair selection process was unfair. 

The science teams generally believed that the same trajectory pair would 
have been selected without the development of the ordinal rankings and the 
utility function values. Nevertheless, five of the science teams believed that 
the usefulness of the ordinal rankings justified the effort required to generate 
them. 
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Finally, the science teams were asked if they would like to see a similar 
analysis performed for critical mission events such as Titan encounters, and 
if they would like to see the analysis repeated in 1977 to select the actual 
trajectory pair to be launched. For both questions there was no agreement 
among the science teams, with the responses ranging from +5 to -5. 

Spearman’s rank correlation coefficient (Ref. 22) was used to test for 
correlation between Question 7: "Was the selected trajectory pair good or 

bad for your science team? 11 and other responses of the science teams, as 
shown in Table VIII. The responses to Question 7 are not correlated at the 
0.05 level of significance with the ordinal rankings or the utility function values 
of the selected trajectory pair by the nine science teams. Even when the 
responses of the Radio Science Team and Science Team 3 are deleted from the 
data, the correlation is still not significant. It must be concluded that several 
of the science teams perceived the selected trajectory pair to be good or bad 
for their science investigations on the basis of criteria other than the ordinal 
rankings or utility function values. 

The responses to Question 7 were correlated at the 0.05 level of 
significance with Questions 10, 14, and 18. Question 10 asked if the selected 
trajectory pair was a good or bad decision in terms of the "science value" 
of the mission. Question 14 asked if the trajectory pair selection process 
was fair. Question 18 asked if the science teams would like to repeat the 
analysis in 1977 to select the trajectory pair to be launched. Thus the opinions 
of the science teams concerning the selection process were correlated with 
whether they perceived the selected trajectory pair to be good or bad for their 
science investigations. 


X. CONCLUSIONS 

The trajectory selection process was successful because of a number of 
factors. By means of the mission constraints levied on the trajectory design, 
it was possible to separate the programmatic issues from the science issues. 
Thus the science teams could be asked to evaluate the trajectory pairs solely 
on the basis of their science preferences, Another important factor was that 
compatible alternatives actually existed -- this is partially fortuitous, but 
more strongly a result of the detailed project planning and coordinated science 
investigation selection process. The collective choice rules were in general 
agreement, reflecting this compatibility. 
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Table VIII 


A Statistical Test for Correlation with Question No. 7 


QUESTION No. 7: WAS THE SELECTED TRAJECTORY PAIR GOOD OR BAD FOR YOUR SCIENCE TEAM? 

r s = SPEARMAN RANK CORRELATION COEFFICIENT 


CORRELATION WITH QUESTION No. 7 

r s 

LEVEL OF 
SIGNIFICANCE 

ORDINAL RANKINGS OF SELECTED TRAJECTORY PAIR BY 9 SCIENCE TEAMS 

0.35 

NOT SIGNIFICANT 
AT 0.05 LEVEL 

ORDINAL RANKINGS OF SELECTED TRAJECTORY PAIR BY 7 SCIENCE TEAMS 
(RSS AND SCIENCE TEAM 3 DELETED) 

0.40 

NOT SIGNIFICANT 
AT 0.05 LEVEL 

UTILITY FUNCTION VALUES OF SELECTED TRAJECTORY PAIR BY 9 SCIENCE 
TEAMS 

0.52 

NOT SIGNIFICANT 
AT 0.05 LEVEL 

UTILITY FUNCTION VALUES OF SELECTED TRAJECTORY PAIR BY 7 SCIENCE 
TEAMS (RSS AND SCIENCE TEAM 3 DELETED) 

0.47 

NOT SIGNIFICANT 
AT 0.05 LEVEL 

QUESTION No. 10: WAS THE SELECTED TRAJECTORY PAIR A GOOD OR BAD 
DECISION IN TERMS OF THE SCIENCE VALUE OF THE MISSION? 

0.91 

<0.01 

QUESTION No. 14: WAS THE TRAJECTORY PAIR SELECTION PROCESS FAIR? 

0.74 

<0.05 

QUESTION No. 18: WOULD YOU LIKE TO REPEAT THE ANALYSIS IN 1977 TO 
SELECT THE TRAJECTORY PAIR TO BE LAUNCHED? 

0.64 

<0.05 
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The generation of the candidate trajectory pairs was an essential part 
of the trajectory pair selection process, even though it lies outside most of 
the formalism presented in this article. The generation of the trajectory 
pairs was an iterative process, requiring the science teams to identify those 
general characteristics of the trajectories which were required by their 
science investigations and requiring the JPL trajectory analysts to construct 
trajectory pairs containing these characteristics. As the process proceeded, 
better trajectory pairs were constructed. Of the original list of 24 trajectory 
pairs which were documented and distributed to the science teams for consider- 
ation, four were dropped, and only four were subsequently ranked in the top 10 
by the rank sum collective choice rule. None were among the last three in 
contention. Finally, as a result of the information exchange between the science 
teams and the JPL trajectory analysts, it was possible to further improve 
the trajectory pair selected from the candidate list. 

With the information generated in the trajectory selection process, an 
improved set of candidate trajectory pairs could now be generated. Certainly 
more trajectory pairs with improved ring occultation characteristics would 
be included, and possibly trajectory pairs could be constructed with high 
rankings by all the science teams. Some trajectory pairs from this set 
could in principle be preferred to the selected trajectory pair. 

The science teams willingly participated in the trajectory selection 
process because they recognized the necessity for them to understand the 
trajectory alternatives and to develop their science investigation requirements, 
and they recognized that if a consensus could be reached among the science 
teams, the Project Manager would accept the science recommendation as the 
Project standard trajectory pair. Also, the science teams had participated 
in the generation of the trajectory pairs, and the set of candidate trajectory 
pairs contained at least one trajectory pair which was considered as very good 
by each science team. 

The science teams were able to adequately express their preferences for 
the candidate trajectory pairs. The ordinal rankings by the science teams were 
essential to this process, and only minor problems were encountered in 
eliciting this information. The cardinalization lotteries and the reasons for 
their use were not endorsed by all the science teams. Nevertheless, all of 
them did submit cardinal values, based either on the cardinalization lotteries 
or on a formula weighted and evaluated for each trajectory characteristic. 
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Since the normalization lotteries did not appear to properly encode the 
preferences of the science teams for their least-preferred trajectories, it 
was necessary to test the sensitivity of the collective choice results to the 
normalization assumptions through the use of two other normalization procedures. 
As was shown in Table IV, whether the utility function values were scaled 
upward from 0.0, p^, or 0.6/0.8 made only minor differences in the rankings 
of the trajectory pairs by the collective choice rules. 

While gaming and coalition-formation did occur, there is no evidence that 
it influenced the trajectory pair selection. Some science teams may have 
assigned low utility function values to a large number of trajectory pairs in 
an attempt to bias the selection process toward a few trajectory pairs with 
specific characteristics. The Radio Science Team (RSS) unfortunately biased 
themselves out of the analysis by assigning to a large number of trajectory 
pairs the minimum value that could be negotiated with the Project. As a 
result, the Radio Science Team had no differential effect on the collective 
choice values of the nine trajectory pairs ranked highest by the rank sum 
collective choice rule. 

One other problem with respect to the science team evaluation of the 
trajectory pairs should be mentioned. Agreement was never achieved on the 
precise criteria to be applied to the evaluation. Three issues never completely 
resolved were: (1) whether the two trajectories of each pair should be considered 
as providing complementary or redundant science, (2) whether the total 
mission or only the encounter aspects of the trajectory pairs should be 
considered, and (3) whether each team should consider the trajectory pairs 
in the narrowest context as satisfying the requirements of their science team, 
or in a broader context of also satisfying the requirements of other science 
teams with complementary objectives. 

Could the selected trajectory pair have been identified without the 
decision analysis formalism? While there is no definitive answer to this 
question, there are two indications that it could not have been. First, the SSG 
was given several opportunities to state a science -preferred trajectory pair, 
and none was forthcoming. Second, the Project and the science teams 
earlier had been working with a trajectory pair developed during the pre- 
ceding year. It had been assumed that this earlier trajectory pair was quite 
satisfactory, and it could have been expected to rank high among the other 
alternatives. This earlier trajectory pair was included on the candidate 
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list as Trajectory Pair 20. It was most surprising to find that Trajectory 
Pair 20 was ranked 28 by the rank sum collective choice rule! 

Clearly the ordinal rankings of the science teams were essential to the 
selection process. These ordinal rankings and the rank sum collective choice 
rule probably would have been sufficient to ultimately identify either Trajectory 
Pair 26 or 31 as the science-preferred trajectory pair. Nevertheless, the 
authors conclude that the cardinal utility evaluation by the science teams 
was an important part of the selection process. The cardinal utility evaluation 
aided the selection between Trajectory Pairs 26, 29, and 31, it tested the 
collective choice analysis for sensitivity to strength of preference not revealed 
by the ordinal rankings, and it permitted a wider range of collective choice 
rules to be used in the analysis. 

In summary, the methodology presented in this article did provide a 
suitable framework for each science team to assess its preferences, and to 
communicate these preferences to the other science teams. The science teams 
were then able to arrive at a consensus in an effective manner and to recommend 
to the Project a science -preferred trajectory pair which was subsequently 
implemented as the Project standard trajectory pair, 
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